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Web Usability Test Findings and Analysis Issues 



Jae Soon An 
Su-Hong Park 
Eun-Ok Back 

Indiana University Bloomington 

Abstract 

In this paper, we discuss findings and data analysis issues that resulted from a case usability test. The findings relate to the 
generation of design ideas. Through usability test, it was found that users have a somewhat unified viewpoint regarding the 
menus on the side areas of a web page. They tend to view them as a ''quick " way to access "specific " and 'frequently searched " 
information. The analysis issues are concerned with analyzing task completion rate, time taken, and path taken to finish the 
tasks. When analyzing these measures, the researcher needs to consider nature of the web site, portal site in this case usability 
test, rather than simply looking at whether and how the test participants found the answer. 



Introduction 

This paper discusses findings and analysis issues derived from a lab usability test The procedure of conducting a lab 
usability test is pretty much standardized and there are standard sets of data typically collected in web site tests such as, task 
completion rate, time taken, paths taken, and verbal protocol. The analysis of the data, however, varies widely depending on the 
case. When analyzing usability test data, it is important to consider the nature of the web site. Depending on the nature of the 
site, the same data can be analyzed differently resulting in different usability indices. In addition, it is well-known in the usability 
literature that deriving design ideas from usability test data is a difficult task. Generating design ideas is closely related to how 
the researcher analyzes test data. This paper describes and discusses how a web design team considered the nature of the portal 
site in data analysis and derived design ideas by distilling data into a finding and the finding into design ideas. 

Redesign of a University Web Site 

The case usability test this paper derives data from was conducted as a part of redesign project that aimed to update the 
existing home page of Indiana University Bloomington (lUB) [2], The home page contained links to the web sites that various 
institutions in the university had already created. Therefore, it functioned as a portal to various web sites and employed 
hierarchical menu structure. 

It was acknowledged that the site had a high degree of usability because a group of graduate students in the university 
conducted a user-centered design research and applied the research results when designing the site in 1995 [1], Although usable, 
there occurred several reasons to redesign the site. First of all, it was reported that prospective students look at universities' home 
pages a lot when they consider applying for schools. Keeping the site up-to-date was important in the view of the university's 
marketing strategy. In addition, information organization of the site was getting messy as the site was accommodating publication 
requests from various institutions in the university over the years. For example, some pages had too many links and some links 
were not placed in the page where users would expect to see them. Finally, there has been a consistent request that the site 
should enhance its visual aspects. With these reasons, lUB Web team began a redesign project in January, 2000 with an 
emphasis on user-centered design principle. 

As the first step of the redesign effort, the design team created a preliminary prototype outlining link categorization only and 
tested usability of the link categorization. Then, the team produced second prototype where they incorporated results of the 
usability test and included visual elements. Although the link categorization was assumed to be usable, the team conducted 
another bigger scale usability evaluation on the second prototype to ensure usability of the overall prototype. Usability 
evaluation on the second prototype had various goals. In addition to collecting general usability measures such as task 
completion rate, time taken to complete the tasks, and the links chosen to complete the tasks, the evaluation aimed to measure 
effectiveness of new navigation features and hear users' opinion about the site in general and visual design in specific. It also 
targeted to assess accessibility and speed of the site. To achieve such various goals, it was necessary to use various test methods 
such as a lab usability test, heuristics evaluations, a web-based survey, foreign font display check, and page loading speed check. 

Lab Usability Test 

The findings and analysis issues discussed in this paper resulted mainly from the lab usability test. The lab test employed 
typical usability test methodology. Participants were selected through purposeful sampling. Nine users of the current lUB home 
page were recruited as test participants considering their occupation, web use, gender, and nationality. In the test, the participants 
were asked to perform eleven tasks using either the new prototype or the existing web site. Through random assignment, 
participants 1, 3, 4, 5, and 9 were asked to use the new prototype, whereas 2, 6, 7, and 8, the existing site. Through the eleven 
tasks, participants were asked to search for typically and frequently searched information in lUB site. The tasks also required the 
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participants to use new navigational features and menu structures. Throughout the test, participants were asked to think out loud. 
The test administrator observed the test session collecting performance measures such as task completion, time taken, and paths 
taken as well as the participants’ comments and emotional expressions. She sometimes prompted and probed think-aloud, and 
responded to the user’s task-related questions. The whole test sessions were videotaped for later review. After the usability test, 
a debriefing interview was conducted. The test administrator solicited further comments about the events during the test that she 
did not understand. In addition, she asked for comments about the web site in general and any suggestions for improvement. 



Data Analysis Considering Nature of the Portal Site 

Task completion rate, time taken, and paths chosen to complete the tasks are three classical usability measures. Typically, 
successful completion is considered to be the point when the participant finds the answer. The time taken and the links chosen 
till the completion point are typically analyzed. Such typical way of analysis, however, didn’t seem to work well in the lUB web 
site test. As the target site of the test was a portal site providing links to various external web sites, it was inevitable for the 
participants to use sites other than the target site (i.e., external sites) in order to finish the tasks. Indeed, some users spent much 
time looking at external pages. Some failed to find the answers due to the design problems of the external sites. Simply looking 
at whether the participant had found the answer or not, and counting total time taken didn’t seem to be appropriate measurements. 
After several rounds of discussion, the team decided to make a distinction between the use of lUB home pages and the use of 
other sites and collect the three measures during the use of lUB home pages only. 

Task Completion 

In the case of task completion rate, the team decided to consider that a participant completed a task successfully if she found 
the link leading to the external site containing the answer (i.e., correct link), regardless she found the answer or not. The fact that 
the user had found the correct link was assumed to indicate that the link categorization in lUB site was effective enough for the 
user to get to the right place, which was the main goal of the site. If the user failed to find the answer after getting to the correct 
page, it was due to the poor design of the external page, which was beyond the goal of the re-design project. Table 2.1 
summarizes task completion rate following this analysis decision. If the user found the correct page but failed to find the answer, 
it was indicated as f/s and counted as a success rather than a fail. Table 2.2 summarizes task completion rate by just considering 
whether the user found the answer or not. Note the difference in the number of people completed the task successfully between 
the two tables. T test result on the difference was t = 1.884, df=10, p=.089. Considering small sample size, the difference was 
almost significant. 



<Table 2J> Task completion rate considering the use oflUB site only 
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f/s: the user found the correct page but could not find the information due to the poor design of Qy, 
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Time Taken 

In the case of the time taken to finish the tasks, the team decided to count the time spent before the users found the correct 
external site for the first time. Here, the word “for the first time” is important because several occasions occurred where users 
found the correct external site initially but could not find the answer in the site. Then, they went back to lUB site, tried other 
external sites, and went back to the correct site which they looked at previously and found the answer eventually. In this case, 
the time taken to find the correct external site for the first time seemed to be an accurate measure of the effectiveness of the link 
categorization. Table 3.1 summarizes time taken to find the correct external page for the first time whereas Table 3.2, total time 
taken to finish the tasks. The mean time taken is significantly (t= -4.3 18, df=lO, p=.002) different depending on the two different 
ways of counting time. 
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<Table 3J> Time taken to find the correct external page for the first time (in second) 
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<Table 3.2> Total time taken to find the answer (in second) 
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Paths 

When analyzing the links the participants have chosen till they find the answer (i,e., path taken), it is difficult to derive 
general currency due to the wide variety of the links people usually choose. It is often possible to analyze how many people 
followed the ideal thread of links and how many did not, or what links in the ideal thread people did not choose correctly. Such 
analysis provides the design team with a general evaluation on the effectiveness of the link categorization. However, they do not 
render much design ideas, other than the requirement to re-place a few links that people did not choose correctly in an ad hoc 
manner. 

The team sought out a more systematic way to utilize the path data and found out that a rich set of design ideas could emerge 
by summarizing the first link the participants chose to complete each task. Table 1 summarizes the first link participants chose to 
complete the tasks in the lUB site test. Almost all participants chose the same link to perform tasks 2, 4, 5, and 9, To perform 
tasks 1, 7, 8, and 10, participants chose more than one link however there was one link that majority of the participants (i.e., three 
out of five) chose. For example, to perform task 7, three out of five users chose “ss” as the first link whereas two users chose 
different links. For tasks 3, 6, and 1 1, people chose three different links and there was no particular link most participants chose. 



<Table J> First link participants chose to complete each task 
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Note: Link names are abbreviated, E,g,) "pg" is "people & groups" and "ss" is "student services." 

By examining the degree of variability in the choice of the first link, the team could generate design ideas. The data indicated 
that information related to the tasks 2, 4, 5, and 9 were categorized intuitively enough for most people to select correct entry 
point. On the contrary, those related to 3, 6, and 1 1 were not. Therefore, information related to the tasks 3, 6, and 1 1 needed to 
be either labeled better or placed in a more appropriate link hierarchy. The data also gave ideas as to where to put certain 
information. For example, the data clearly indicated that users expected to find information related to the task 8 either in "ar" or 
"ss" page. Unfortunately, the prototype did not provide that information in either of the pages. 
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Deriving Design Ideas On Side Menus 

The first level page of the lUB site contained ten main menus and each menu had three descriptors underneath. In addition to 
the main menus, the page included more items on the right-hand side and the upper part of the page, which the design team called 
“side menus.” The existing web site did not have side menus. Figure 1 and 2 depict such difference in the menu structure of the 
first level page between the new prototype and the existing site. 



<Figure 1 > First level page 's menu structure in the new prototype 
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<Figure 2> > First level page’s menu structure in the existing site 
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The idea of having side menus in addition to main menus arose from the team’s desire to maximize the use of screen 
space. There have been research articles reporting that some people do not scroll down while they browse menus on the page. In 
addition, researchers like Jakob Nielsen have been arguing for the breath of menu over the depth. 

Initially, the team did not have a clear understanding about the nature of the menus appropriate to each group of the 
menus. Through usability tests and interviews with test participants, however, it was found out that users had a certain and 
unified viewpoint on the side menus. They tended to view them as a “quick” way to access “specific” and “frequently searched” 
information. The specific and frequently searched information are the ones highly identifiable to most users of the site. In the 
case of a university web site, access points to e-mail system, grading system, or commonly used knowledge base are some 
examples. This finding derived from the interviews and the observations of the paths test participants took while performing the 
tasks. Firstly, participants requested that the information presented through the side menus should be replicated somewhere in 
the main menus. They wanted to be sure that the information was available to them even if they would ignore the side menus 
entirely. This indicated that the main function of the side menus was a way to access certain information quickly rather than a 
primary provider of information. Secondly, the prototype subjected to the usability test had a couple of side menus whose nature 
was somewhat general such as “how to apply” and “computing help.” To find admission and computer help information, 
participants used main menus rather than side menus because, they didn’t know what specific information they needed to find out 
yet and wanted to ensure they were browsing comprehensive list of information available in the site. This indicated that the 
information presented through the side menus should be specific. 

Having identified the nature of information appropriate for the side menus, the design team then discussed kinds of 
information and a naming strategy appropriate for the side menus. By examining keyword search statistics of the existing lUB 
site, the team found out that three most specific and frequently searched items were e-mail, INSITE, and Knowledge Base. 
INSITE was a student service system through which students could view or update their address, class schedules, grades, tuition 
and fee bills, etc. Knowledge Base was a database of university computing help. As for the menu names, the team adopted the 
actual names of the systems. The general public might not understand what the names meant however people belonging to the 
community would understand them. Since the side menus were frequently searched information, targeting people belonging to 
the community seemed to be appropriate. 
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Conclusion 

Analyzing usability test data to fulfill true purpose of the usability test is not always straightforward. Design teams conduct 
usability tests usually to measure the degree of usability of their design (i.e., usability indices) and gather ideas for improvement 
(i.e., design ideas.) This paper described how lUB web team analyzed test data to meet such purposes. In conclusion, it is 
important to consider nature of the design when analyzing test data. In addition, the process of distilling findings from data and 
linking the findings to design ideas is an important part of analysis. 
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